WIP: Relocation based build (inplace free) #445

angerman · 2017-11-01T13:30:01Z

This is essentially a reorganization on how libraries and ghc are built. The idea is to produce
ghc builds in _build/stage2/{bin,lib}, that could just be packed up and shipped around. This
also means there should be no more inplace files left.

This also includes a few additional changes:

gets rid of the Evac_thr, Scar_thr logic, that is not present in the Make base build system
make the cabal file parser a bit more powerful, such that it can configure packages, (it does not run the proper configure logic yet).

Things that do now yet work:

There are no wrapper scripts.
ghc --interactive fails to run, due to libgmp relocation issues
Note this was due to hadrian relying on builing the in-tree-gmp on macOS. I believe
this is equally broken in the make based build system.
No target prefixes yet. (e.g. arch-vendor-os-ghc).

Noteworthy:

Requires patches to Cabal (Add s/cmm-sources + extra-bundled-libraries haskell/cabal#4857, basedir haskell/cabal#4874, Adds virtual-modules to handle cases like GHC.Prim from ghc-prim. haskell/cabal#4875)
Note: this is now an collection of serval PRs, which are included in my zw3rk/cabal/work branch.
Requires changes to GHC (D4121, D4135, D4174, D4175, D4176)

Others:

The value of this has definitely been that I now know more about hacking on Hadrian.

NOTE: I do not expect this to be merged now. I want this to be more a basis
for discussion. I would have tried to make the same idea work with the make based
build system, but as Hadrian is the supposed way forward, I did not intend to waste
much time on the old(?) build system. I am going to pursue this direction further.

TODO:

Package Install / Registration Targets (stage+1). E.g. a library built with stage0 should be registers in the packageDb at stage1. Similar to how Programs built in stage0 end up in stage1/bin. This will also remove the need to copy the package.conf.d over from stage1 to stage2 to obtain the full stage2 compiler.
Reinstate wrapper rules
Drop the custom bootstrap db logic. (it's just the stage1 db then).
Reinstate the Install rules.
Check running the test suite.
Check building documentation.
Roll more of ghc-cabal into hadrian.

This allows to run it during the build without the wrapper.

Again, if ghc-pkg is run without a wrapper, we need to provide it with the flag.

No more /inplace/ logic.

This might need some rethinking, as we might want to rebuild them per stage (or copy?) We could e.g. build them in stage0, and just copy them into stage1 and stage2.

This again should probably be done by having the package depend on the right files. ghc depends on settings, llvm-targets, platformConstants and the ghc/ghci-usage.txt files. hsc2hs depends on the template.

hsc depends on the template-hsc.h, and ghc depends on settings, platformConstants, llvm-targets, ghc-usage.txt, ghci-usage.txt. As such they will bring the files in on their own, and the copy target logic is no longer required.

E.g. buld ghc-cabal with stage0, it's part of the stage1 then.

…/feature/reloc # Conflicts: # src/Base.hs # src/Builder.hs # src/GHC.hs # src/Hadrian/Builder.hs # src/Rules/Documentation.hs # src/Rules/Install.hs # src/Settings/Builders/Hsc2Hs.hs # src/Settings/Builders/RunTest.hs

Types.X modules become X.Type

…/feature/reloc # Conflicts: # src/GHC.hs

alpmestan · 2018-02-18T19:30:35Z

(Just adding a link as some sort of "TODO", for me to revisit my documentation rules patches to add --quickjump: ghc/ghc@9ff4cce)

snowleopard · 2018-02-18T19:46:26Z

@alpmestan Oops, sorry, I've already rushed ahead with #505 to make sure I won't forget this.

alpmestan · 2018-02-18T22:13:59Z

No worries :)

This commit implements two significant changes (that were not easy to separate): - Don't use ghc-cabal anymore for getting information about Haskell packages. We now instead directly use Cabal-the-library. - Make the build root configurable. This effectively gets rid of the inplace logic and allows us to place _all_ build artefacts in some directory of our choice, by passing '--build-root <some path>' to hadrian. The code for this was mostly taken from snowleopard#445.

…le (#531) * Use Cabal directly in place of ghc-cabal; make build root configurable. This commit implements two significant changes (that were not easy to separate): - Don't use ghc-cabal anymore for getting information about Haskell packages. We now instead directly use Cabal-the-library. - Make the build root configurable. This effectively gets rid of the inplace logic and allows us to place _all_ build artefacts in some directory of our choice, by passing '--build-root <some path>' to hadrian. The code for this was mostly taken from #445. * fix documentation rules * remove some leftover unrelated, commented-out code * more documentation fixes, address some feedback * cleanup * more cleanup * boot and configure explicitly in travis CI scripts * update cabal/ghc versions in .travis.yml (8.0.x not supported anymore) * temporarily disable dynamic ways in Settings.Default * update appveyor script * travis: when booting with 8.2.2, build a complete stage2 compiler * Fix CI? Try to fix the CI by adding the `debug` rts way back in. * Update Quickest.hs Replicate the make build systems build flavours. * Update .travis.yml - Run selftest, and build in separate instances. - try with python2 - and unify mac to stage2 * Update .travis.yml upgrade python on mac * [travis] os x: test the freshly built ghc * Get rid of two unused GhcCabalMode constructors * fix ghc-split rule, get rid of Install/Wrappers rules * address more feedback * ConfiguredCabal -> PackageData, more comments, more feedback addressed * make the complete stage 2 build the default * use a dummy package instead of base in Rules.hs * update CI scripts * attempt at fixing hadrian's -c option * .travis.yml: use -c everywhere again * travis: back to explicit './boot && ./configure' * update README.md and doc/user-settings.md to reflect configurable build root * some more feedback

izgzhen · 2018-04-04T09:58:19Z

I’m curious what is left to be done in this PR after #531?

alpmestan · 2018-04-09T12:01:43Z

I’m curious what is left to be done in this PR after #531?

I think we're pretty much done, aren't we @angerman ? We got rid of the need for ghc-cabal, made the build root configurable, incorporated a bindist rule (it's even cleaner than the code that I wrote in this branch).

snowleopard · 2018-04-09T13:57:06Z

@alpmestan Sounds great! Once you are confident nothing has been lost, please close this PR.

angerman · 2018-04-17T10:05:31Z

Guess we can close this for now. If I'll end up with issues building my cross compilers again, I'll probably start a new branch.

snowleopard · 2018-04-17T14:48:52Z

@angerman Once again, many thanks for this amazing PR :-)

angerman added 30 commits October 25, 2017 16:15

Move package.conf.d into $build/$stage/lib

f10e62b

[hsc2hs] provide with staged template

c4a297b

[ghc-pkg] provie the --global-package-db flag

1f76250

This allows to run it during the build without the wrapper.

[ghc-cabal] Pass down the --global-package-db flag to ghc-pkg

713617c

Again, if ghc-pkg is run without a wrapper, we need to provide it with the flag.

test

cc3ca47

[Programs] install in $stage/bin

e955497

No more /inplace/ logic.

dead install for now.

315b632

More elaborate generate logic.

b0359ed

dead install

1636836

[ghc] no more stages, no more inplace

500336f

[Builders] most of them are in stage1/bin now, instead of stage0.

c99ef5b

This might need some rethinking, as we might want to rebuild them per stage (or copy?) We could e.g. build them in stage0, and just copy them into stage1 and stage2.

[base] less in-place logic.

0ecb089

[Rules] Copy rules for multiple stages

89bd0fe

This again should probably be done by having the package depend on the right files. ghc depends on settings, llvm-targets, platformConstants and the ghc/ghci-usage.txt files. hsc2hs depends on the template.

[hsc2hs] Staged, and dependent on template-hsc.h

523139e

[hsc2hs,ghc] drop inplace copy targets.

e41f939

hsc depends on the template-hsc.h, and ghc depends on settings, platformConstants, llvm-targets, ghc-usage.txt, ghci-usage.txt. As such they will bring the files in on their own, and the copy target logic is no longer required.

buildPath -> contextPath

1afd1c2

Build ContextPath rewrite

88a2a2c

reinstante /build folder

3206db3

Full package names; and build dir.

2fcbde2

stage munging.

a2805b6

[ghc-cabal] commands configure/copy/reg

8d43007

contextInstallDir is libDir

8ba790d

[libraries] proper pkgid

6977e5f

[pkg-db] unify naming.

e096710

[ghc-cabal] rules for copy and register

966e0c2

Paths

7442787

Use the ghc-cabal from the next stage.

361f4b6

E.g. buld ghc-cabal with stage0, it's part of the stage1 then.

copy and register library logic

d2a2833

[ghc-pkg] Add clone (describe + register) logic.

7b38480

Logic for asking builders.

afb84ba

angerman added 4 commits February 13, 2018 14:35

iserv/libiserv logic

f7db9b4

Adds -fllvmng flag

e3d27af

Profing flag

9505159

Flags.

f0a39fd

alpmestan mentioned this pull request Feb 15, 2018

move a bunch of types into dedicated modules #502

Merged

angerman and others added 4 commits February 15, 2018 18:55

Types.X modules become X.Type

025197c

Merge pull request #4 from alpmestan/angerman/feature/reloc

40709cc

Types.X modules become X.Type

Merge branch 'master' of github.com:snowleopard/hadrian into angerman…

709c178

…/feature/reloc # Conflicts: # src/GHC.hs

This was referenced Feb 25, 2018

Improve documentation for cross compilation #513

Open

Better organize source tree to separate policy from mechanism #347

Open

Adds QuickCrossNCG flavour.

b5d8e2d

snowleopard mentioned this pull request Mar 12, 2018

Error: at least ftruncate or _chsize functions are required to build #523

Closed

alpmestan mentioned this pull request Mar 19, 2018

Use Cabal directly in place of ghc-cabal + make build root configurable #531

Merged

alpmestan mentioned this pull request Mar 22, 2018

Implement installation and binary/source distribution rules #219

Closed

3 tasks

angerman mentioned this pull request Mar 25, 2018

Add support for llvm toolchain / code gen #178

Closed

angerman closed this Apr 17, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Relocation based build (inplace free) #445

WIP: Relocation based build (inplace free) #445

angerman commented Nov 1, 2017 •

edited

alpmestan commented Feb 18, 2018

snowleopard commented Feb 18, 2018 •

edited

alpmestan commented Feb 18, 2018

izgzhen commented Apr 4, 2018

alpmestan commented Apr 9, 2018

snowleopard commented Apr 9, 2018

angerman commented Apr 17, 2018

snowleopard commented Apr 17, 2018

WIP: Relocation based build (inplace free) #445

WIP: Relocation based build (inplace free) #445

Conversation

angerman commented Nov 1, 2017 • edited

This also includes a few additional changes:

Things that do now yet work:

Noteworthy:

Others:

TODO:

alpmestan commented Feb 18, 2018

snowleopard commented Feb 18, 2018 • edited

alpmestan commented Feb 18, 2018

izgzhen commented Apr 4, 2018

alpmestan commented Apr 9, 2018

snowleopard commented Apr 9, 2018

angerman commented Apr 17, 2018

snowleopard commented Apr 17, 2018

angerman commented Nov 1, 2017 •

edited

snowleopard commented Feb 18, 2018 •

edited